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An  Adaptive  Inpainting  Algorithm  Based  on  DCT 
Induced  Wavelet  Regularization 

Yan-Ran  Li^,  Lixin  Shen^,  and  Bruce  W.  Suter^ 


Abstract — In  this  work,  we  propose  an  image  inpainting  opti¬ 
mization  model  whose  objective  functional  is  a  smoothed  norm 
of  the  weighted  non-decimated  discrete  cosine  transform  (DCT) 
coefficients  of  the  nnderlying  image.  By  identifying  the  objective 
fnnction  of  the  proposed  model  as  a  sum  of  a  differentiable 
term  and  a  non-differentiable  term,  we  give  a  basic  algorithm 
inspired  by  Beck  and  Tebonlle’s  recent  work  [1]  for  the  proposed 
model.  Based  on  this  basic  algorithm,  we  propose  an  automatic 
way  to  determine  the  weights  involved  in  the  model  and  update 
them  at  each  iteration.  The  discrete  cosine  transform  as  an 
orthogonal  transform  is  used  in  various  applications.  We  view 
the  rows  of  a  discrete  cosine  transform  matrix  as  the  filters 
associated  with  a  multiresolution  analysis.  Non-decimated  wavelet 
transforms  with  these  filters  are  explored  to  analyze  Images  to 
be  Inpainted.  Our  numerical  experiments  verify  that  under  the 
proposed  framework,  the  filters  from  a  discrete  cosine  transform 
matrix  demonstrate  promise  for  the  task  of  image  inpainting. 

Index  Terms — Discrete  Cosine  Transform,  Framelet,  Morean 
envelope.  Inpainting,  minimization. 

1.  Introduction 

In  many  applications,  only  partial  data  are  available  in 
an  image  due  to  a  variety  of  reasons  including  impulsive 
noise  caused  by  malfunctioning  pixels  in  camera  sensors, 
faulty  memory  locations  in  hardware,  or  transmission  in  a 
noisy  channel,  text  or  signature  superposed  on  an  image, 
and  a  scratch  in  a  picture  [2],  [3].  Recovery  of  missing 
pixels  is  called  image  inpainting,  an  active  area  of  research 
in  image  processing.  Applications  of  image  inpainting  include 
old  films  restoration,  video  inpainting  [4],  de-interlacing  of 
video  sequences  [5],  and  cloud  removal  from  remotely  sensed 
images  [6].  Restoring  missing  pixels  becomes  indispensable 
in  the  above  applications. 

Many  successful  algorithms  for  image  inpainting  have  been 
developed  in  the  past  decade.  These  inpainting  algorithms 
can  be  roughly  classified  into  three  groups:  geometric  partial 
differential  equation  (PDE),  patch,  and  sparse  representation 
based  ones.  Classical,  yet  popular,  methods  for  image  inpaint¬ 
ing  use  PDEs  and  variational  formulations  to  propagate  avail¬ 
able  image  information  from  observed  domains  into  missing 
regions  as  a  way  of  smoothly  transporting  the  contours  of 
the  image  into  the  regions  being  inpainted  [3],  [7],  [8],  [9], 
[10],  [11],  [12].  Within  the  category  of  PDE-based  methods, 
there  are  a  number  of  approaches  which  perform  well  for 
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piecewise  smooth  images  with  sharp  edges.  Patch-based  (or 
exemplar-based)  inpainting  methods  fill  in  missing  pixels  of 
an  image  from  known  non-local  observed  data  by  exploring 
local  repetitions  of  local  information.  Recent  work  in  this 
direction  includes  [13],  [14],  [15],  [16],  [17]  and  the  references 
therein.  Algorithms  in  this  category  propagate  the  known 
patches  into  missing  patches  gradually  in  a  fashion  of  cut- 
and-paste,  and  perform  well,  especially  for  missing  regions 
with  texture  and  large  sizes.  Sparse  representation  for  image 
inpainting  was  recently  addressed  in  [18],  [19],  [20],  [21], 
[22],  [23],  [24],  [25],  [26],  [27],  [28],  and  the  references 
therein.  The  idea  of  these  algorithms  is  to  sparsely  represent 
an  image  by  a  redundant  system  which  is  formed  by  a  set 
of  transforms  such  as  the  discrete  cosine  transform,  wavelets, 
framelets,  and  curvelets.  The  missing  pixels  are  then  inferred 
by  shrinking  coefficients  adaptively  and  iteratively  from  this 
sparse  representation. 

In  this  paper,  we  propose  an  inpainting  method  which  could 
be  viewed  as  a  sort  of  combination  of  a  variational  approach 
and  a  sparse  representation  based  approach.  The  proposed 
inpainting  method  is  developed  based  upon  an  optimization 
model  whose  objective  functional  is  a  smoothed  £i  norm  of  the 
coefficients  of  the  underlying  image  under  a  given  redundant 
system.  The  redundant  system  is  generated  from  the  discrete 
cosine  transform  (DCT)  matrix  of  second  type  [29].  More 
precisely,  we  identify  the  rows  of  a  given  DCT  matrix  as 
the  filters  associated  with  a  multiresolution  analysis  which  is 
referred  to  as  the  DCT-Haar  wavelet  system  in  the  following 
discussion.  The  non-decimated  transform  using  the  DCT-Haar 
wavelet  filters  yields  a  redundant  system  that  is  used  in  our 
model.  We  show  that  this  redundant  system  incorporated  with 
the  proposed  model  performs  particularly  well  for  inpainting 
images  with  incomplete  information.  We  propose  an  adaptive 
inpainting  algorithm  to  solve  the  proposed  model.  We  also 
propose  a  way  to  automatically  tune  the  parameters  appearing 
in  the  algorithm. 

The  remainder  of  this  paper  is  organized  as  follows.  In 
Section  II,  we  begin  with  reviewing  several  variational  inpaint¬ 
ing  models  based  on  sparse  representations.  We  then  propose 
our  model  for  inpainting  which  is  regularized  by  DCT-Haar 
wavelets.  In  Section  III,  we  develop  an  adaptive  iterative  algo¬ 
rithm  for  solving  the  proposed  model.  Numerical  experiments 
for  the  proposed  algorithm  are  presented  in  Section  IV.  Our 
conclusions  are  drawn  in  Section  V. 
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II.  An  Inpainting  Model  Regularized  by  DCT-Haar 
Wavelets 


Our  inpainting  model  in  this  paper  is  formulated  as  an  opti¬ 
mization  problem  in  which  the  variational  objective  functional 
has  a  regularization  term  formed  by  a  sparse  representation  of 
the  underlying  image.  We  point  out  that  the  proposed  optimiza¬ 
tion  model  is  connected  with  the  reweighted  minimization 
model  [30],  but  with  several  distinct  and  promising  properties. 

We  begin  with  introducing  some  notation  used  in  the  fol¬ 
lowing.  Let  the  original  image  forg  be  defined  on  the  domain 
n  =  {1,2, ...  ,n}  and  the  nonempty  proper  subset  2?  of  be 
given.  The  observed  image  g  is  modeled  as 


forg[k],  k&n\V-, 

h[k\,  k  GV, 


(1) 


where  h[k]  with  k  GV  could  represent  any  types  of  degrada¬ 
tions  to  the  original  image  including  impulsive  noise  and  texts 
superposed  on  forg- 

Associated  with  the  sets  and  V,  we  define  an  n  x  n 
diagonal  matrix,  denoted  by  Pxi,  whose  fc-th  diagonal  entry  is 
1  if  A:  G  n  \  2?  and  0  if  fc  G  2?.  The  goal  of  image  inpainting 
is  to  seek  an  image  /  such  that  Pu/  =  Pvg  while  /  can 
truthfully  retain  original  information  of  forg- 

Prior  to  introducing  our  variational  model  for  image  in¬ 
painting,  we  review  some  sparse  representation  based  image 
inpainting  models  which  are  most  related  to  ours. 


A.  A  Brief  Review  of  Three  Inpainting  Models 

The  first  example  among  these  models  is  the  morphological 
component  analysis  based  simultaneous  cartoon  and  texture 
image  inpainting  [18].  The  minimization  task  in  [18]  is 

{/c:/t*}  =  argmin|i||Pi,(6f-  /c-  /t)|li  -f  A||lLc/c|li 

fcjt  I  ^ 

+  A||m/dli+7ll/c||Ty},  (2) 

where  A  and  7  are  regularization  parameters,  Wc  is  a  redun¬ 
dant  system  leading  to  sparse  representations  for  cartoon-like 
images,  Wt  is  a  redundant  system  able  to  represent  texture¬ 
like  images  sparsely,  and  ||  •  \\tv  is  the  total  variation.  The 
image  /*  -f  ff  will  be  the  inpainted  outcome.  A  work  similar 
to  the  one  in  [18]  was  also  formulated  in  [22]. 

The  second  example  is  from  [31],  [20],  [23]  using  tight 
framelets.  The  corresponding  minimization  task  is 

X*  =  argmin|i||Pi,(5  -  VL^a;)||^ 

+  l||(Id- W»||2  +  ||ra;|li|,  (3) 

where  IL  is  a  tight  framelet  system  with  W^W  =  Id  and  T 
is  a  diagonal  matrix  with  non-negative  diagonal  components. 
The  inpainted  image  reads  as  Pj^g  -f  (Id  —  Pti)W^ x* .  The 
numerical  experiments  in  [31],  [20],  [23]  showed  that  model 
(3)  is  particularly  suitable  for  cartoon-like  images. 

The  third  example  from  [28]  is  an  extension  of  the  first  two 
examples  for  tackling  images  having  both  cartoon  and  texture 


contents.  The  proposed  model  is 


=  aTgmml-\\P-Dig-  ^k)\\ 


fcG{c,t} 


E  m-WkWj)xkm+  E 


1 1  > 


(4) 


fcG  {c,t} 


where  Wc  and  Wt  are  framelet  systems  favoring  cartoon 
and  texture  contents  of  images,  respectively,  Tc  and  T*  are 
diagonal  matrices  with  non-negative  diagonal  components. 
The  inpainted  image  resulting  from  model  (4)  is  P-ogP  (Id  — 
P-d){WJx*  +  WJx{). 

The  success  of  models  (2)  and  (4)  fully  relies  on  the 
selection  of  appropriate  redundant  systems  Wc  and  Wt-  The 
rationale  of  choosing  desirable  Wc  and  Wt  is  that  Wc  should 
yield  sparse  representations  for  the  cartoon  parts  of  the  un¬ 
derlying  image  while  Wt  should  yield  sparse  representations 
for  the  texture  parts  of  the  image.  However,  two  issues  arise 
from  using  models  (2)  and  (4).  The  first  issue  is  that  there  is  a 
lack  of  precise  mathematical  definitions  of  cartoon  and  texture 
components  of  an  image.  This  leads  to  a  difficulty  in  separating 
an  image  exactly  into  its  cartoon  part  and  texture  part.  Hence, 
it  affects  the  choice  of  appropriate  systems  Wc  and  Wt-  The 
second  issue  is  that  the  regularization  parameter  A  in  (2)  and 
the  diagonal  matrices  and  Tj  in  (4)  are  essential,  but  should 
be  pre-determined.  An  automatic  way  of  determining  A,  Tc 
and  Tj  is  not  available  yet.  Both  issues  need  to  be  addressed 
if  models  (2)  and  (4)  are  adopted  for  image  inpainting. 

Instead  of  directly  addressing  the  aforementioned  two  is¬ 
sues,  we  take  a  different  point  of  view  about  image  inpainting. 
As  usual,  we  infer  each  missing  pixel  on  T>  using  the  known 
information  from  its  neighborhood.  Our  approach  is  to  choose 
a  redundant  system  associated  with  a  multiresolution  analysis 
such  that  the  filters  from  the  multiresolution  analysis  has  the 
ability  to  extract  information  from  neighborhoods  of  pixels 
with  different  sizes.  The  candidates  of  redundant  systems 
are  associated  with  multiresolution  analysis  generated  from 
the  DCTs  that  will  be  presented  in  detail  in  the  following 
subsection. 


B.  DCT-Haar  Multiresolution  Analysis 

Discrete  cosine  transforms  are  frequently  used  orthogo¬ 
nal  transforms  in  applied  mathematics  and  engineering  [29]. 
Among  various  types  of  the  DCTs,  the  DCT-II  is  the  most 
popular  one  of  all  and  will  be  chosen  below  in  our  discussion. 
The  standard  m  x  m  DCT-II  matrix  C  is  given  by 


Cfc  cos 


(fc  -  l)(2j  -  l)7r 
2m 


k,j 


1,2,. ..,m  , 


where  ei  =  1  and  =  s/2  for  k  =  2,3, ...  ,m.  The  matrix 
C  is  orthogonal,  i.e.,  C^C  =  I.  Denoting  by  Ck  the  fc-th  row 
of  we  can  directly  verify  that  the  sum  of  entries  of  Cfc 

is  1  for  k  =  1  and  zero  for  k  =  2,3, . . .  ,m.  Hence,  the  vector 
Cl  can  be  viewed  as  a  low-pass  filter  while  the  others  can  be 
viewed  as  high-pass  filters.  In  particular,  when  m  =  2,  ci  and 
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C2  are  low-pass  and  high-pass  filters  corresponding  to  the  well- 
known  Haar  multiresolution  analysis.  Indeed,  ci  and  Ck,  k  = 
2,3, ...  ,m,  from  a  general  mxm  DCT-II  matrix,  are  low-pass 
and  high-pass  filters  associated  with  a  multiresolution  analysis 
of  dilation  m,  which  is  called  the  DCT-Haar  multiresolution 
analysis  in  the  following  discussion. 

We  denote  by  Pk  the  matrix  representation  of  the  filters 
Ck,  k  =  1,2, ...  ,m,  under  a  proper  boundary  condition.  For 
a  simple  exposition,  we  illustrate  the  construction  of  the  Pk 
under  the  periodic  condition.  In  this  scenario,  the  matrix  Pk 
of  size  f  X  f  is  a  circulant  matrix  with  [ck  Oix{e-Tn)]  as  its 
first  row  and  is  written  as  follows: 

Pfe  =  circulant  ( [cfc  Oix(f_m)]).  (5) 

Proposition  1:  Let  C  be  the  mxm  DCT-II  matrix  and  Ck 
be  the  fc-th  row  of  -^C.  For  the  matrices  Pk  defined  by  (5), 
we  have  that 

m 

J2pjPk  =  u. 

k^l 

Proof:  Let  us  denote  by  the  first  column  of  the  matrix 


Pk,  fovk  =  1,2,. . . 

,  m.  From  the  orthogonality  of  C,  we  have 

that 

m 

= 

k^l 

f  P,  if/?  =  lor/3>f'-TO  +  l; 

[  0,  otherwise. 

(6) 

m 

'^Pk[lS\pk 

[r]  =0  for  all  1  <  j5  <  t  <  1. 

(7) 

Let  U  be  the  £  x  £  Fourier  transform  matrix,  i.e.,  U  := 

=  1,2,...,£].  Since  any 
circulant  matrix  can  be  diagonalized  by  the  Fourier  transform 
matrix,  we  have  that 

m  /  ra  \ 

L  ^kPk  =^U*  diag{U*pk)diag{Upk)  j  U. 

k=l  \fc=l  / 

By  the  equation  U*U  =  Id,  we  just  need  to  show 
di8'S(^*Ffc)diag(C/pfe)  =  jid.  It  suffices  to  prove 

i]r=i  =  I  for  sach  a  G  {1,2,  ...,£}. 

Let  us  denote  by  Ua  the  ath  column  of  U.  Note  that 
\{U*pk)[a\\^  =  pJuaKpk  =  ELi  + 

2Ei</3<T<rffe(^a[/3]<[T])pfc[/3]pfc[T].  Summing  this  equa¬ 
tion  for  k  from  1  to  m,  interchanging  the  order  of  the 
summations  EfeLi  ELi  Ei<^<T<r’  ^rid  using 
equations  (6)-(7)  together  with  the  fact  of  |Ma[/?]|  =  ^  for  all 
a  and  /3,  we  get  that  EEi  =  j-  This  completes 

the  proof.  ■ 

For  a  given  one-dimensional  vector  x  G  a  vector  of 
PkX  is  the  non-decimated  DCT-Haar  wavelet  coefficients  of 
X  produced  by  the  filter  Cfc.  From  the  coefficients  PkX,  k  = 
1,2, ...  ,m,  the  vector  x  can  be  perfectly  reconstructed  from 
PkX  due  to  Proposition  1. 

The  two  dimensional  DCT-Haar  multiresolution  analysis, 
constructed  by  the  one  dimensional  DCT-Haar  multiresolution 
analysis  through  the  tensor  product  technique,  will  be  used 
to  represent  images.  The  matrices  Qi  •=  Pi  ®  Pi 

=  Pi®  Pj  with  i,j  =  l,2,...,m  and  {i,j)  ^ 
(1, 1)  are,  respectively,  the  low -pass  and  high-pass  filters 


associated  with  the  two  dimensional  DCT-Haar  multiresolution 
analysis.  Here,  the  symbol  ®  denotes  matrix  tensor  product. 
For  an  lx  I  image  x,  we  can  view  it  as  a  vector  (denoted  it 
by  X  again)  in  M"  with  n  =  by  concatenating  its  columns. 
The  vectors  QkX,  k  =  1,2, ...  ,m'^,  are  the  non-decimated 
DCT-Haar  wavelet  coefficients  of  x  and  satisfy  the  perfect 
reconstruction  formula,  i.e.,  x  =  'ET=iQkQkx. 

With  these  matrices  Qk,  we  form  a  matrix  W  of  the  size 
of  m^n  X  n  as  follows 

W:=[Qj,Qj,...,Ql,X.  (8) 

By  Proposition  1,  we  have  the  perfect  reconstruction  condition 
for  the  matrices  Qk,  that  is,  W^W  =  Id.  Therefore,  IF  is  a 
tight  framelet  transform  matrix. 


C.  The  Proposed  Inpainting  Model 

The  proposed  inpainting  model  with  a  redundant  system  IF 
generated  from  the  DCT  is  as  follows: 

miii{^||IF/-c;||i  +  ||rd||i  :  P-d!  =  Pvg^,  (9) 

where  F  is  a  diagonal  matrix  with  non-negative  diagonal 
components  and  d  is  an  auxiliary  vector. 

Let  us  explain  model  (9)  in  detail.  The  requirement  Pp/  = 
Ppp  says  that  the  solution  to  be  found  should  interpolate  the 
observed  data  g  exactly  on  the  set  H  C  P,  in  other  words,  the 
solution  of  model  (9)  is  in  the  set 

C:={/:/gK”  and  Pj,f  =  Pj,g},  (10) 

which  is  convex.  In  the  set  C,  the  solution  /  to  be  sought  is 
sparse  in  the  sense  that  its  transformed  coefficients  IF/  are 
close  to  a  sparse  vector  d  with  respect  to  the  weight  matrix  F. 

Model  (9)  can  be  written  into  a  compact  and  unconstrained 
form  by  using  the  notion  of  an  indicator  function  and  the 
Moreau  envelope  in  convex  analysis.  The  indicator  function 
be  on  the  convex  set  C  is  defined  as 

0,  if  /  G  C; 

+00,  otherwise. 

Then,  the  objective  functional  in  model  (9)  becomes 
min  {^||IF/  -  d\\l  +  ||r(i||i  +  ic(/)|  • 

2 

Next,  for  a  proper  convex  function  tp  defined  on  R'"  ",  that  is, 
(p  :  R™  ”  — >•  Ru{+oo},  having  a  non-empty  domain  (the  set 
on  which  p  is  finite),  the  Moreau  envelope  [32]  of  p,  denoted 
by  env^,  is  a  function  from  R'"  "  to  R,  defined  for  a  given 
u  G  R™""  by 

env„(M)  :=  min  L||u  -  v\\\  +  p{v) 

V  [  ^ 

Notice  that 

minj^llIF/- d||i  +  ||rd||i  +  rc(/)| 

=  mm|mm|^||IF/-d||^  +  IlFdllij  +ic(/)|  . 
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By  identifying  W f,  d ,  and  ||  •  ||  loF  as  w,  z;,  and  (f,  respectively, 
in  the  definition  of  the  Moreau  envelope,  model  (9)  is  written 
as 

mm{env||.||jor(W)  +  '-c(/)}  •  (11) 

As  reported  in  many  research  articles  (e.g.  [18],  [22], 
[33]),  the  DCT-Haar  multiresolution  analysis  is  suitable  for 
representing  images  rich  in  texture  information.  We  would  like 
to  demonstrate  that  the  DCT-Haar  multiresolution  analysis  is 
also  suitable  for  representing  images  rich  in  cartoon  informa¬ 
tion  provided  that  the  resulting  coefficients  of  the  underlying 
image  are  properly  modified  and  an  appropriate  nonlinear 
reconstruction  approach  is  used.  To  this  end,  we  illustrate 
this  point  by  a  numerical  example  portrayed  in  Figure  1.  An 
algorithm  for  model  (11)  will  be  developed  in  section  III  while 
more  examples  will  be  provided  in  section  IV.  Figure  1(a)  is  an 
original  cartoon  image  while  Figure  1(b)  is  the  cartoon  image 
overlaid  with  text.  The  inpainted  images  shown  in  Figure  1(c)- 
(f)  with  the  peak  signal-to-noise  ratio  (PSNR)  values  of 
37.38dB,  37.37dB,  37.04dB  and  39.96dB  are,  respectively, 
from  three  competing  algorithms,  namely,  spline  framelets 
(SF)  based  on  [20],  spline  framelet  and  local  DCT  (SF-LDCT) 
based  on  [28],  and  morphological  component  analysis  (MCA) 
based  on  [18],  and  our  proposed  algorithm.  We  can  see  that 
the  quality  of  the  image  in  Figure  1(f)  is  better  than  those 
images  in  Figure  l(c)-(e)  in  terms  of  visual  perception. 

In  the  rest  of  this  section,  we  make  a  connection  between 
model  (11)  and  the  weighted  minimization  model  for  sparse 
signal  recovery  in  [30]. 

The  problem  discussed  in  [30]  is  to  determine  an  object 
xq  G  R*  from  the  data  y  =  $a;o,  where  $  is  a  n  x  g  matrix 
with  n  <  q.  Under  the  sparsity  assumption  on  the  xq  which 
we  wish  to  recover,  it  is  common  in  compressive  sensing  to 
solve  the  combinatorial  optimization  problem 

mm  ||a;||o  subject  to  y  =  (12) 

where  ||a;||o  is  the  number  of  non-zero  elements  of  x.  It  is 
well  known  that  the  optimization  problem  (12)  is  nonconvex 
and  generally  impossible  to  solve.  A  common  alternative  is 
to  replace  ||x||o  in  problem  (12)  by  ||a;||i.  As  a  result,  it 
converts  the  nonconvex  optimization  problem  to  a  convex  one. 
The  use  of  the  norm  in  sparse  signal  recovery  has  been 
well  understood  and  justified  (see,  e.g.,  [34],  [36]  and  the 
references  therein).  Motivated  by  the  observation  that  larger 
coefficients  are  penalized  more  heavily  in  the  norm  than 
smaller  coefficients,  unlike  the  more  democratic  penalization 
of  the  norm,  it  was  proposed  in  [30]  to  consider  the 
“weighted”  optimization  problem 

min  ||Ta;||i  subject  to  y  =  $a;,  (13) 

sGR'’ 

where  T  is  the  diagonal  matrix  with  positive  numbers 
f[l],  f[2], . . . ,  f[g]  on  the  diagonal.  In  [30],  a  heuristic  way  was 
suggested  to  solve  the  problem  (13)  as  follows:  (i)  beginning 
with  the  initialized  weight  matrix  of  all  I’s  on  its 

diagonal;  (ii)  solving  =  argmin^p^,  ||T(^^x||i  subject  to 
y  =  <i>x;  (iii)  updating  the  weights  ■=  + 

e)“^  for  k  =  1,2,  ...,q  and  a  given  positive  number  e; 


and  (iv)  terminating  on  convergence  or  £  attaining  a  specified 
maximum  number  of  iterations;  otherwise  incrementing  £  and 
going  to  step  (ii). 

Our  proposed  model  (11)  is  in  the  sprit  of  (13),  but  with 
several  additional  promising  properties.  Let  us  explain  it  now 
in  detail.  Define  x  :=  W f,  y  :=  P^g,  and  $  :=  PxiW^ , 
where  W  is  of  size  q  x  n.  We  consider  a  problem  of  seeking 
a  sparse  set  of  coefficients  x  that  synthesizes  the  signal  /  = 
X  as  follows: 


minenv||.||jor(a:)  subject  to  y  =  (14) 


which  is  identical  to  problem  (11)  when  W  is  an  orthogonal 
system.  By  the  definition  of  the  Moreau  envelope,  a  direct 
calculation  gives,  for  a  nonnegative  number  a. 


enVo|.|(M) 


a|M|  —  if  |u|  >  a; 

otherwise. 


(15) 


Since  F  is  an  n  x  n  diagonal  matrix  with  non-negative  diagonal 
elements,  then  the  function  ||  •  ||i  o  F  is  separable.  Hence 
env||.||jor(a;)  =  ZLi  env,y[fe]|.|  (x[fc]).  Further,  by  using  for¬ 
mula  (15),  we  define  a  diagonal  matrix  T  with  diagonal 
elements  as 


7R  2\x[k]\  ) 

i|x[fc]| 


if  |x[fc]|  >  7[fc]; 
otherwise. 


(16) 


With  this  weight  matrix  T,  we  have  that  env||.|| jor(a:)  = 
||rx||i.  This  is  the  reason  why  we  call  F  the  weight  matrix 
as  well.  By  comparing  the  formulation  (14)  together  with 
(16)  to  that  of  (13),  one  can  view  (14)  as  a  weighted  £^ 
minimization  problem.  The  strategy  in  determining  the  weight 
T  is  obviously  different  from  the  one  given  in  [30].  In 
Step  (iii)  of  the  aforementioned  approach  in  [30],  the  weight 
corresponding  to  an  entry  of  a  sparse  signal  is  simply  the 
reversal  of  the  sum  of  the  magnitude  of  the  entry  and  a  small 
positive  number  e.  The  weight  for  an  entry  of  a  sparse  signal 
in  (16)  is  determined  by  a  pre-determined  threshold,  i.e.,  the 
matrix  F  that  will  be  estimated  in  an  automatic  way  in  the 
next  section.  On  the  other  hand,  unlike  the  objective  functional 
in  (13),  the  one  in  (14)  is  differentiable  and  its  gradient  is 
Lipschitz  continuous.  This  property  is  particularly  important  in 
developing  a  fast  and  efficient  numerical  algorithm  for  solving 
the  optimization  problem  (14).  Again,  this  will  be  illustrated 
in  the  next  section. 


III.  Algorithms 

In  this  section,  we  propose  an  iterative  algorithm  for  solving 
model  (11)  which  involves  a  redundant  system  W  generated 
from  the  DCT  and  a  weight  matrix  F.  A  variant  of  the 
algorithm  with  a  varying  weight  matrix  F  in  iterations  is 
suggested. 

A.  Algorithm  for  Model  {IV) 

This  subsection  presents  a  fast  algorithm  for  solving 
model  (11).  The  cost  function  of  the  model  is  the  sum  of 
the  functions  env||.||jor  o  W  and  lc-  By  the  definition  of 
indicator  function,  lq  is  not  differentiable.  We  will  show 
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(d)  (e)  (f) 

Fig.  1.  Performance  of  various  inpainting  algorithms  for  a  cartoon  image  with  text,  (a)  the  original  test  image;  (b)  the  test  image  with  text;  inpainted  images 
by  (c)  SF  (PSNR=37.38  dB);  (d)  SF-LDCT  (PSNR=37.37  dB);  (e)  MCA  (PSNR=37.04  dB);  and  (f)  the  proposed  algorithm  (PSNR=39.96  dB),  respectively. 


that  env||.||jor  o  W  is  differentiable  and  its  gradient  is  Lip- 
schitz  continuous.  Hence,  Beck  and  Teboulle’s  accelerated 
proximal  gradient  (APG)  algorithm  [1]  is  suitable  for  solving 
model  (11). 

The  accelerated  proximal  gradient  algorithm  is  developed 
for  a  general  nonsmooth  convex  optimization  model 

mm{V’(/)  +  ^(/)},  (17) 

where  (j)  :  K"  — >■  KU{+oo}  is  a  convex  function,  t/>  :  R"  — >■  R 
is  continuously  differentiable  with  Lipschitz  continuous  gra¬ 
dient  L(t/>),  i.e.,  ||VV’(/i)  -  VV’(/2)||2  <  L{i’)\\fi  -  /2II2 
for  every  /i,/2  G  R”.  The  corresponding  APG  algorithm 
proposed  in  [1]  is  given  in  Algorithm  1. 


Algorithm  1  (Accelerated  proximal  gradient  (APG)}: 

1)  set  n  =  1,  /o  =  P-ng  +  (Id  —  Pv)X,  where  X  is 
a  matrix  whose  entries  are  uniformly  distributed  on 
the  interval  [0,  255]; 

2)  Take  ui  =  /o  and  L  =  L{ip)  as  a  Lipschitz  constant 
of  X'lp; 

3)  For  fc  =  1,  2, . . .,  compute 

a)  fk  =  a.Tgmmj{(j){f)  +  f||/  - 

{uk  -  j-y^juk))  nil 

b)  Tfc+I  =  - ^^2 - - 

C)  Uk+1  =  fk+  (^)  ifk  -  fk-l) 


To  apply  APG  Algorithm  1  for  model  (11),  we  need  to 
identify  the  corresponding  functions  ^  and  </>  in  model  (11).  To 
this  end,  we  give  some  properties  of  the  functions  env||.||jor  o 
W  and  Lc  in  model  (11). 


Lemma  1:  If  T  is  a  diagonal  matrix  with  nonnegative  diag¬ 
onal  entries  and  W  is  the  tight  framelet  transform  matrix  given 
in  (8),  then  env||.||jor  oW  is  continuously  differentiable  and 
its  gradient  is  Lipschitz  continuous  with  the  Lipschitz  constant 
1. 

Proof:  It  is  well-known  that  the  Moreau  envelope 

of  any  function  is  always  differentiable  and  the  gradient  of 
the  Moreau  envelope  is  Lipschitz  continuous  with  Lipschitz 
constant  1  [32],  [37].  Thus,  the  chain  rule  yields  V(env||.||joro 
W)  =  VL^Venv||.||j^or  °  W.  Therefore,  for  any  fi  and  /2  in 
R”,  we  have  that 


||V(env||.||^or  o  W)(fi)  -  V(env||.||^or  o  W^)(/2)||2 
<  l|W^^I|2||H^||2||/i-/2||2. 

Notice  that  ||IL^|||  =  IlFLili  =  =  1  due  to 

W^W  =  Id.  Hence,  the  Lipschitz  constant  of  the  gradient 
V(env||.||jor  o  W)  is  1.  This  completes  the  proof.  ■ 

The  following  result  explains  how  to  compute  Venv||.||jor- 
Lemma  2:  If  L  is  an  n  x  n  diagonal  matrix  with  non¬ 
negative  diagonal  entries,  then  the  fcth  entry,  k  = 
of  the  gradient  of  env||.||jor  at  /  G  R"  is 


Venv||.||^or(/)[fc] 


7feSign(/[fc]),  if  |/[fc]|  >  7fc; 
f[k],  otherwise. 


where  jk  is  the  fcth  diagonal  entry  of  L. 

Proof:  Since  L  is  a  diagonal  matrix  with  non¬ 

negative  diagonal  elements,  then  we  have  that  env||.||j^or(/)  = 
®'av|.yj,.|  (/[fc]).  Hence,  the  conclusion  of  the  lemma 
follows  immediately  from  (15).  ■ 

Lemma  3:  If  g,  as  shown  in  (1),  is  an  image  to  be  inpainted 
on  the  domain  V  and  the  convex  set  C  is  given  in  (10),  then 
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for  any  a  >  0  and  /  €  R"  we  have  that 
P-vg  +  (Id  -  Pv)f  =  argmin  l^llu  -  f\\l  +  tc(M)}  (18) 


Proof:  By  the  definition  of  the  indicator  function,  we 
know  that 


^\\u-m+cc{u) 


ih-fWh  ifueC; 

+00,  otherwise. 


Because  PxiCid  —  Pj))  =  0,  then  llu— /llo  =  WPviu  —  f)\\2  + 
II (Id  -  Pv)iu  -  /)||^  >  \\Pvi9  -  f)\\l  for  any  u  €  C.  The 
equality  in  the  above  holds  when  we  choose  u  =  P-j^g  +  (Id  — 
P'D)f  ■  This  completes  the  proof.  ■ 

With  Lemmas  1  and  3,  we  identity  the  functions  env||.||joro 
W  and  Lc  in  model  (11)  as  ip  and  (p  in  model  (17),  respec¬ 
tively.  By  using  Lemma  1-3,  an  algorithm  based  on  APG  for 
model  (11)  is  proposed  in  Algorithm  2. 


Algorithm  2  (Algorithm  for  Model  (11)).' 

1)  set  Ti  =  1,  /o  =  P-vg  +  (Id  —  Pxi)X,  where  X  is 
a  matrix  whose  entries  are  uniformly  distributed  on 
the  interval  [0,  255],  and  a  pre-determined  matrix  L; 

2)  Take  ui  =  /o; 

3)  For  fc  =  1,  2, . . .,  compute 

a)  fk  =  Pvg  +  (Id  -  P'D){uk  - 


b) 


W'Xenv\\.\\^or{Wuk)) 
_  l  +  Vl+4r| 

T^k+1  —  2 


C)  Uk+1  =  fk+  (^)  ifk  -  fk-l) 


Let  {fk}  be  the  sequence  generated  by  Algorithm  2.  Then  all 
elements  in  the  sequence  are  in  C.  Let  /*  be  the  point  at  which 
the  function  env||.||jor(II^/)  +  '^c(/)  in  model  (11)  achieves  its 
minimal  value.  Then  f*  must  be  in  C  as  well.  For  Algorithm  2, 
the  rate  of  convergence  of  the  sequence  of  function  values 
env||.||jor(II^/fc)  to  the  optimal  value  env||.||jor(II^/*)  is 
given  as  follows: 

env||.||^or(IT"/fc)  -  env||.||^or(II^/* )  < 

for  all  k  >  1.  The  proof  of  the  above  estimate  is  given  in  [1]. 

The  remaining  problem  for  Algorithm  2  is  how  to  choose 
the  matrix  F.  We  plan  to  use  the  procedure  proposed  in  our 
recent  work  [38]  to  automatically  estimate  F  associated  with 
evaluating  env||.|por(ITM)  for  an  input  image  u.  To  this  end, 
let  us  denote  by  v  the  Wu.  For  any  non-negative  integer  p  and 
positive  integer  q,  we  denote  by  quot  (p,  q)  and  rem  (p,  q)  the 
quotient  and  the  remainder,  respectively,  when  p  is  divided  by 
q.  For  the  fth  entry  of  v,  we  can  identify  this  entry  by  the 
triplets  {ii,£2,^3)  where  £i  =  quot  (quot  (^  —  l,n),m)  +  1, 
£2  =  rem(quot(^  —  1,  n),  m)  -f  1,  and  £3  =  rem(f  —  1,  n)  -f  1. 
The  pair  of  indexes  (£i,£2)  indicates  that  v[£]  the  £th  entry 
of  V  is  from  the  subband  generated  by  Pi^  0  Pi^  and  the 
index  £3  indicates  that  this  entry  v[£]  is  the  fath  entry  of  the 
vector  ®  ^^2)'^-  With  this  notation,  as 

suggested  in  [38],  ji  the  fth  entry  of  F  is  estimated  as  follows: 


0, 

V2al 

m‘^av[£]  ’ 


if  1  <  £  <  n; 
otherwise. 


(19) 


where  is  the  noise  variance  of  the  image  u  and  is 
the  variance  of  v[£]  with  the  assumption  of  v\£]  being  Laplace 
distributed.  Here,  can  be  estimated  by  [39],  [38] 


vie] 


=  max  ■ 


E 

k&miv[e]) 


V^\v[k]\ 

mvm 


2  -1  A  — 6 


where  the  index  set  ff{{v[£])  is  the  neighborhood  of  the 
coefficient  v\£\  in  the  corresponding  (('1,^2)  subband  and 
|lH(t;[£])|  is  the  cardinality  of  the  set  d\(v[£]).  The  window 
iR{v[£])  of  size  (m-|-2)  x  (to-|-2)  is  adopted  in  our  experiments. 

Based  on  the  formula  (19),  a  variant  of  Algorithm  2  with 
adaptive  updating  the  matrix  F  in  iterations,  is  described  in 
Algorithm  3. 


Algorithm  3  (Adaptive  Algorithm  for  Model  (11)).' 

1)  Set  Ti  =  1,  /o  =  P-ng  +  (Id  —  P'd)X,  where  X  is 
a  matrix  whose  entries  are  uniformly  distributed  on 
the  interval  [0,  255],  and  a  pre-given  positive  integer 
s; 

2)  Take  ui  =  fo; 

3)  For  k  =  1,2, . . compute 


a)  Estimate  F^  from  the  input  image  Uk'  based 
on  (19)  with  k'  =  s  ■  quot  (fc  —  1,  s)  -f  1 

b)  fk  =  Pvg  +  (Id  -  Pv){uk  - 


IF^Venv||.||  orjVFufc)) 

c)  Tfc+i  =  iVpM 

d)  Uk+i  =  fk+  (^)  ifk  -  fk-i) 


We  remark  that  7^  being  zero  for  1  <  f  <  n  implies  that 
the  scaling  coefficients  of  Wuk  (i.e.,  Qinf)  in  Step  3(a)  of 
Algorithm  3  will  be  unmodified.  Furthermore,  the  matrices  F^, 
for  s((7— 1)-|-1<A:<s(7  with  q  being  a  natural  number,  are 
identical  and  are  essentially  estimated  from  the  Us(q_i)_|_i  via 
(19).  In  other  words,  the  matrices  F^  are  updated  only  once 
for  every  s  iterations.  In  particular,  if  s  =  1,  the  matrices  F^ 
will  be  updated  at  each  iteration. 


IV.  Experiments 

In  this  section,  we  present  numerical  results  to  demonstrate 
the  performance  of  our  proposed  inpainting  algorithm.  We 
shall  compare  results  of  the  proposed  algorithm  to  those  of 
three  state-of-the-art  methods  in  the  literature. 

In  our  numerical  experiments,  three  test  images  are  “Lena”, 
“Cameraman”,  and  “Barbara”  with  a  size  of  256  x  256. 
Three  benchmark  schemes  are  used  in  our  experiments:  Spline 
Eramelet  based  (SE)  [20],  Spline  Eramelet  and  Local  DCT 
based  (SE-LDCT)  [28],  and  Morphological  Component  Anal¬ 
ysis  based  (MCA)  [18].  The  implementations  of  all  three 
algorithms  are  provided  by  their  own  authors.  Our  proposed 
algorithm  in  the  experiments  is  Algorithm  3  which  is  essen¬ 
tially  parameter- free.  In  the  spirit  of  reproducible  research, 
the  Matlab  source  codes  of  the  proposed  Algorithm  3  can 
be  obtained  by  sending  an  email  to  lshen03@syr.edu. 
Each  algorithm  is  carried  out  until  the  stopping  condition 
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ll/fe+i  -  /fc|li/||/fe||i  <  5  X  10  '‘is  satisfied.  The  quality  of 
the  inpainted  images  is  evaluated  in  terms  of  the  peak  signal- 
to-noise  ratio  (PSNR)  defined  by 

PSNR=  lOlogio^ - 

WJrestored  JorgW 

where  frestored  is  the  restored  image  with  a  total  of  n  pixels. 

This  section  contains  three  subsections.  In  the  first  two  sub¬ 
sections,  the  scenarios  for  text  removal  and  random  impulsive 
noise  removal  will  be  discussed.  The  last  subsection  is  about 
the  effect  of  choosing  the  DCT-Haar  multiresolution  analysis 
of  various  dilations  m. 

A.  Experimental  Results  on  Text  Removal 

Removing  texts  superposed  on  images  is  a  typical  problem 
in  image  inpainting.  Two  text  documents  named  as  “Text  1” 
and  “Text  2”  are  overlying  on  the  test  images  shown  in  the  first 
row  and  the  second  row  of  Figure  2,  respectively.  The  words 
in  the  “Text  2”  are  thicker  and  denser  than  that  in  “Text  1”. 
A  set  consisting  the  locations  of  the  words  in  “Text  1”  and 
“Text  2”  serves  as  the  set  V. 

The  first  experiment  tests  numerically  the  choice  of  the 
number  s  in  Algorithm  3  in  terms  of  the  CPU  time  that  the 
algorithm  consumed  and  the  PSNR  values  of  the  inpainted 
images  for  text  removal.  We  observe  from  Table  I  that  the 
algorithm  with  s  being  1  and  8  performs  comparably  in  terms 
of  the  PSNR  values  of  the  inpainted  images,  but  the  algorithm 
with  s  =  8  uses  much  less  the  CPU  time  than  that  with  s  =  1. 
Hence,  in  the  rest  of  the  paper,  we  always  choose  s  =  8  for 
Algorithm  3. 

TABLE  I 

The  pair  (■,  ■)  represents  the  PSNR  value  of  the  inpainted 

IMAGES  AND  THE  CPU  TIME  (SECONDS)  USED  FOR  TEXT  REMOVAL  BY 

Algorithm  3  with  the  parameter  s  being  1  and  8,  respectively. 


s 

“Lena” 

“Cameraman” 

“Barbara.” 

Case 

1 

(34.16,  203) 

(31.96,  213) 

(37.15,  162) 

“Text  1” 

8 

(33.95,  76) 

(31.96,  81) 

(36.98,  64) 

1 

(29.41,  446) 

(28.55,  368) 

(30.37,  438) 

“Text  2” 

8 

(29.31,  144) 

(28.60,  135) 

(30.12,  142) 

As  expected,  for  each  algorithm,  the  inpainted  image  for  an 
image  with  “Text  1”  should  be  better  than  that  with  the  “Text 
2”  in  terms  of  PSNR  values.  This  can  be  clearly  observed  from 
Table  II.  By  comparing  the  performance  of  Algorithm  3  with 
the  other  methods  SF,  SF-LDCT,  and  MCA,  the  PSNR  values 
listed  in  Table  II  indicate  that  Algorithm  3  is  the  best  in  terms 
of  PSNR  values.  We  further  point  it  out  that  the  CPU  times 
consumed  by  both  Algorithm  3  and  the  MCA  are  comparable 
for  all  tested  cases. 

Let  us  have  a  closer  look  at  the  visual  quality  of  the 
inpainted  images  by  various  algorithms.  When  “Text  1”  is 
overlying  on  the  test  images,  the  zoomed-in  portions  of  the 
inpainted  images  are  depicted  in  Figure  3.  One  can  see  that 
Algorithm  3  can  efficiently  remove  the  superposed  words  over 
the  regions  having  both  cartoon  and  texture  components  (see 
Figure  3(Rl-f)-(R3-f)).  Those  words  are  identifiable  in  the 
inpainted  images  by  the  other  three  methods  (see  columns  3, 
4,  and  5  of  Figure  3).  We  can  draw  the  same  conclusion  for 


the  images  having  the  “Text  2”.  The  corresponding  results  are 
shown  in  Figure  4. 

TABLE  II 

The  pair  (■,  ■)  represents  the  PSNR  value  of  an  inpainted  image 
AND  the  CPU  TIME  (SECONDS)  USED  BY  SF,  SF-LDCT,  MCA,  AND 
Algorithm  3  for  text  removal. 


Algorithms 

“Lena” 

“Cameraman” 

“Barbara.” 

Case 

SF 

SF-LDCT 

MCA 

Algorithm  3 

(32.12,  143) 
(31.73,  260) 
(32.48,  92) 
(33.95,  76) 

(30.29,  151) 
(30.58,  282) 
(30.63,  88) 
(31.96,  81) 

(32.81,  127) 
(35.22,  266) 
(35.37,  89) 
(36.98,  64) 

“Text  1” 

SF 

SF-LDCT 

MCA 

Algorithm  3 

(28.36,  228) 
(28.10,  425) 
(28.20,  109) 
(29.31,  144) 

(26.78,  221) 
(26.83,  426) 
(26.47,  110) 
(28.60,  135) 

(27.79,  183) 
(29.27,  415) 
(29.25,  123) 
(30.12,  142) 

“Text  2” 

B.  Experimental  Results  on  Random  Impulsive  Noise  Removal 

The  problem  of  removing  random  impulsive  noise  can  be 
viewed  as  one  of  image  inpainting.  We  adopt  the  adaptive 
median  filter  [40]  to  locate  the  pixels  of  images  corrupted 
by  random  impulsive  noise.  The  detected  locations  form  a 
set  V  which  is  the  region  of  an  image  to  be  inpainted 
and  therefore  is  assumed  to  be  known  in  our  experiments. 
Three  impulsive  noise  levels,  namely  30%,  50%,  and  70%, 
are  used  to  test  robustness  and  efficiency  of  the  proposed 
algorithm.  The  experimental  results  with  SF,  SF-LDCT,  MCA, 
and  Algorithm  3  are  reported  in  Table  III.  As  we  can  observe 
from  this  table,  the  improvements  made  by  Algorithm  3  over 
the  other  three  algorithms  in  terms  of  PSNR  values  are  up  to 
5.31  dB,  which  is  significant. 

TABLE  III 

The  pair  (■,  ■)  represents  the  PSNR  value  of  the  inpainted 
IMAGES  AND  THE  CPU  TIME  (SECONDS)  USED  FOR  BY  SF,  SF-LDCT, 
MCA,  AND  Algorithm  3  for  removing  random  impulsive  noise. 


Algorithm 

“Lena” 

“Cameraman” 

“Barbara” 

Case 

SF 

SF-LDCT 

MCA 

Algorithm  3 

(28.44,  316) 
(27.69,  577) 
(27.85,  92) 
(28.72,  90) 

(25.45,  303) 
(25.23,  540) 
(25.02,  91) 
(26.13,  102) 

(26.13,  264) 
(28.37,  559) 
(29.11,  92) 
(30.35,  95) 

70% 

SF 

SF-LDCT 

MCA 

Algorithm  3 

(31.34,  205) 
(30.70,  372) 
(31.09,  84) 
(32.35,  62) 

(28.92,  198) 
(28.24,  356) 
(28.37,  90) 
(29.97,  74) 

(29.77,  185) 
(32.60,  320) 
(32.67,  85) 
(35.08,  57) 

50% 

SF 

SF-LDCT 

MCA 

Algorithm  3 

(34.96,  147) 
(34.01,  246) 
(35.33,  106) 
(36.50,  50) 

(32.50,  150) 
(31.55,  251) 
(32.19,  106) 
(33.60,  56) 

(34.49,  142) 
(37.30,  264) 
(37.99,  108) 
(39.33,  43) 

30% 

The  visual  comparisons  of  the  inpainted  images  by  SF,  SF- 
LDCT,  MCA,  and  Algorithm  3  are  presented  for  the  images  of 
“Lena”,  “Cameraman”,  and  “Barbara”  corrupted  by  impulsive 
noise  with  noise  level  of  70%.  Figure  5  shows  the  noisy 
“Lena”,  “Cameraman”,  and  “Barbara”  images  while  Figure  6 
presents  the  inpainted  images  using  these  four  algorithms.  It 
can  be  observed  that  all  the  algorithms  can  inpaint  the  images 
well,  but  Algorithm  3  is  the  best  of  them  in  terms  of  the 
inpainted  images  (see  the  last  column  in  Figure  6)  having  less 
ringing  artifacts  than  the  ones  by  SF-LDCT  and  MCA  (see 
the  second  and  third  columns  in  Figure  6)  and  having  a  more 
fruitful  texture  than  the  one  by  SF  (see  the  first  column  in 
Figure  6). 
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(d)  (e)  (f) 


Fig.  2.  The  test  images  superposed  by  “Text  1”  (the  first  row)  and  the  “Text  2”  (the  second  row).  The  square  region  in  each  image  is  the  part  of  the 
corresponding  image  zoomed  in. 


(Rl-a)  (Rl-b)  (Rl-c)  (Rl-d)  (Rl-e)  (Rl-f) 


(R2-a)  (R2-b)  (R2-c)  (R2-d)  (R2-e)  (R2-f) 


(R3-a)  (R3-b)  (R3-c)  (R3-d)  (R3-e)  (R3-f) 


Fig.  3.  Column  1:  Zoomed-in  portions  of  the  original  test  images;  Column  2:  Zoomed-in  portions  of  the  original  test  images  with  “Text  1”;  Columns  3-6 
are  inpainted  images  by  using  SF,  SF-LDCT,  MCA,  and  Algorithm  3,  respectively. 
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(R2-e) 


(R3-e) 


(R2-f) 


(R3-f) 


Fig.  4.  Column  1:  Zoomed-in  portions  of  the  original  test  images;  Column  2:  Zoomed-in  portions  of  the  original  test  images  with  the  “Text  2”;  Columns 
3-6  are  inpainted  images  by  using  SF,  SF-LDCT,  MCA,  and  Algorithm  3,  respectively. 


Fig.  5.  The  images  of  “Lena”,  “Cameraman”,  and  “Barbara”  (from  left  to  right)  corrupted  by  impulsive  noise  with  noise  level  of  70%. 


For  a  further  visual  comparison,  we  present  the  results  for 
the  zoomed-in  portions  of  the  images  shown  in  Figure  6.  These 
results  with  our  proposed  algorithm  together  with  other  three 
competing  algorithms  are  provided  in  Figure  7.  We  can  see  that 
our  proposed  algorithm  retrieves  fine  structures  of  the  original 
images  very  well  and  yields  the  inpainted  images  with  good 
visual  quality.  The  zoomed-in  portion  of  the  “Lena”  image 
(see.  Figure  7(Rl-e))  displays  a  part  of  Lena’s  hair  and  a  part 
of  her  hat.  The  hair  and  edges  of  the  hat  in  the  inpainted 
images  in  Figure  7(Rl-a)-(Rl-c),  which  are  from  SF,  SF- 
LDCT,  and  MCA,  respectively,  are  blurry  with  some  white 
and  black  spots.  In  a  contrast,  the  hair  and  edges  of  the  hat 
in  Figure  7(Rl-d)  from  Algorithm  3  are  more  focused.  For 
the  image  of  “Cameraman”,  the  right  ear  of  the  Cameraman 
(see  Figure  7(R2-e))  is  distinguishable  in  Figure  7(R2-d)  by 
Algorithm  3,  but  are  not  clear  in  the  inpainted  images  in 
Figure  7(R2-a)-(R2-c),  which  are  from  SF,  SF-LDCT,  and 
MCA,  respectively.  The  face  of  the  Cameraman  with  Algo¬ 
rithm  3  looks  smoother  than  the  ones  with  the  other  three 
algorithms.  The  shapes  of  the  camera  in  Figure  7(R2-b)  and 


(R2-d)  are  closer  to  the  original  shape  in  Figure  7(R2-e)  than 
the  ones  in  Figure  7(R2-a)  and  (R2-c).  The  ‘Barbara’  image 
having  both  texture  and  cartoon  components  is  considered  as 
a  good  example  in  image  inpainting.  The  head  scarf  of  the 
Barbara  recovered  by  Algorithm  3  (see  Figure  7(R3-d))  is 
well  preserved  and  visually  looks  the  same  as  the  original 
one  (see  Figure  7(R3-e)).  Noticeable  artifacts  are  shown  in  the 
inpainted  images  (Figure  7(R3-a)-(R3-c))  with  SF,  SF-LDCT, 
and  MCA. 

C.  Ejfects  of  The  DCT-Haar  Wavelets  with  Different  Sizes 

The  reported  performance  of  Algorithm  3  in  the  previous 
two  subsections  is  based  on  DCT-Haar  wavelet  from  the  7  x 
7  DCT-II  matrix.  We  would  like  to  investigate  the  effect  of 
changing  sizes  of  the  DCT-II  matrices  on  the  performance 
of  Algorithm  3.  To  this  end.  Algorithm  3  with  the  DCT-Haar 
transform  matrix  W  from  the  DCT-II  matrix  sizes  5x5,  7x7, 
9  X  9,  and  11  X  11  is  applied  for  inpainting  the  images  in 
the  first  row  of  Figure  2.  The  corresponding  PSNR  values  of 
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Fig.  6.  The  inpainted  images  for  the  images  of  “Lena”,  “Cameraman”,  and  “Barbara”  (from  top  to  bottom)  corrupted  by  impulsive  noise  with  noise  level 
of  70%.  The  results  (from  left  to  right)  are  produced  by  SF,  SF-LDCT,  MCA,  and  Algorithm  3,  respectively. 


the  inpainted  images  are  listed  in  Table  IV.  It  can  be  seen 
that  the  larger  the  size  of  the  DCT-II  matrix,  the  higher  the 
PSNR  value  of  the  inpainted  image.  The  inpainted  images  are 
visually  similar  and  therefore  will  not  be  shown  here.  To  avoid 
compromising  the  performance  and  computational  complexity 
of  Algorithm  3,  the  DCT-II  matrix  size  7  x  7  is  a  suitable 
choice. 

TABLE  IV 

The  pair  (■,  ■)  represents  the  PSNR  value  of  an  inpainted  image 

AND  THE  CPU  TIME  (SECONDS)  BY  ALGORITHM  3  WITH  DIFFERENT  SIZES 
OF  THE  DCT-HAAR  SYSTEMS  FOR  THE  TEST  IMAGES  WITH  “TEXT  1”. 


DCT5 

DCT7 

DCT9 

DCTll 

Image 

(33.42,  29) 
(31.35,  32) 
(35.36,  26) 

(33.95,  76) 
(31.96,  81) 
(36.98,  64) 

(34.19,  169) 
(32.29,  181) 
(37.59,  153) 

(34.35,  364) 
(32.41,  373) 
(37.96,  314) 

“Lena” 

“Cameraman” 

“Barbara” 

V.  Conclusions 

In  this  paper,  we  proposed  an  image  inpainting  model  which 
was  formulated  using  the  smoothed  and  weighted  norm  of 
the  coefficients  of  the  underlying  image  and  the  redundant 
systems  generated  by  the  DCT-Haar  multiresolution  analysis. 
We  developed  an  algorithm  based  on  Beck-TebouIIe’s  method. 
The  weight  matrix  in  the  proposed  algorithm  was  updated 
based  on  the  local  statistics  of  the  coefficients  of  the  solution 


at  each  iteration.  The  algorithm  outperforms  the  three  state- 
of-the-art  alternatives  for  various  examples  in  terms  of  both 
PSNR  values  and  visual  quality  of  the  experimental  results.  We 
claimed  that  the  redundant  system  generated  by  the  DCT-Haar 
multiresolution  analysis  and  the  resulting  inpainting  model 
are  particularly  suitable  for  inpainting  problems  including 
impulsive  noise  removal  and  filling  missing  information  over 
regions  with  moderate  sizes. 
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(R2-e) 


(R3-e) 

Fig.  7.  The  results  for  the  zoomed-in  portions  of  the  noisy  images  shown  in  Figure  6.  The  first  four  columns  are  the  inpainted  images  for  the  zoomed-in 
portions  by  SF,  SF-LDCT,  MCA,  Algorithm  3,  respectively.  The  last  column  shows  the  original  portions  of  the  images. 


(Rl-e) 
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